Communication Characteristics and Hybrid MPI/OpenMP Parallel Programming on Clusters of Multi-core SMP Nodes

Hybrid MPI/OpenMP and pure MPI on clusters of multicore SMP nodes involve several mismatch problems between the parallel programming models and the hardware architectures. Measurements of communication characteristics between cores on the same socket, on the same SMP node, and between SMP nodes on several platforms (including Cray XT4 and XT5) show that machine topology has a significant impact on performance for all parallelization strategies and that topology awareness should be built into all applications in the future. We describe potentials and challenges of the dominant programming models on hierarchically structured hardware. Case studies with the multizone NAS parallel benchmarks on several platforms demonstrate the opportunities of hybrid programming. 1. Mainstream HPC architecture Today scientists who wish to write efficient parallel software for high performance systems have to face a highly hierarchical system design, even (or especially) on “commodity” clusters (Fig. 1 (a)). The price/performance sweet spot seems to have settled at a point where multi-socket multi-core shared-memory compute nodes are coupled via high-speed interconnects. Inside the node, details like UMA (Uniform Memory Access) vs. ccNUMA (cache coherent Non-Uniform Memory Access) characteristics, number of cores per socket and/or ccNUMA domain, shared and separate caches, or chipset and I/O bottlenecks complicate matters further. Communication between nodes usually shows a rich set of performance characteristics because global, nonblocking communication has grown out of the affordable range. This trend will continue into the foreseeable future, broadening the available range of hardware designs even when looking at high-end systems. Consequently, it seems natural to employ a hybrid programming model which uses OpenMP for parallelization inside the node and MPI for message passing between nodes. However, there is always the option to use pure MPI and treat every CPU core as a separate entity with its own address space. And finally, looking at the multitude of hierarchies mentioned above, the question arises whether it might be advantageous to employ a “mixed model” where more than one MPI process with multiple threads runs on a node so that there is at least some explicit intra-node communication (Fig. 1 (b)–(d)). It is not a trivial task to determine the optimal model to use for some specific application. There seems to be a general lore that pure MPI can often outperform hybrid, but counterexamples do exist and results tend to vary with input data, problem size etc. even for a given code [1]. This paper discusses potential reasons for this; in order to get optimal scalability one should in any case try to implement the following strategies: (a) Reduce synchronization overhead (see Sect. 3.5), (b) reduce load imbalance (Sect. 4.2), (c) reduce computational overhead and memory consumption (Sect. 4.3), and (d) Minimize MPI communication overhead (Sect. 4.4). There are some strong arguments in favor of a hybrid model which tend to underline the assumption that it should lead to improved parallel efficiency as compared to pure MPI. In the following sections we will shed some light on Cray User Group 2009 Proceedings SMP node SMP node (a) (c) (d) (b) Node interconnect Socket 1